Skip to main content

Incident Report: No Deviation Alerts on Blast

Date: 2024-03-14
Time: 12:25 (UTC+5:30)
Duration: 10 minutes

Description​

There were no deviation alerts on the chain Blast on Opsgenie and also Grafana dashboard did not show the data for Blast for deviation.

Root Cause​

The Telemetry Airseeker also has a list of chains (hardcoded) and blast wasn't included in it when the team integrated it.

Impact​

There is no data on the Grafana dashboard to check the deviation and we are not getting a deviation alert on Opsgenie which is a potential issue.

Timeline​

  • 12:25 - First noticed the issue.
  • 12:31 - Initial diagnosis.
  • 17:11 - Started the fix.
  • 17:54 - Issue resolved.

Lessons Learned​

It's important to add any steps needed to be done during chain integration no matter how trivial it may be as it may cause potential threats.

Actions Taken​

  • First Bedirhan fixed the outage issue.
  • Vekil identified that Telemetry Airseeker has a list of chains that are hardcoded and fixed it to complete the chain integration.
  • Benjamin added this step in the chain integration guide to avoid the problem next time.
  • Slack escalation link.

Incident Reviewer(s)​

  • Burak
  • Bedirhan
  • Arda
  • Warren Anderson
  • Ali
  • Vekil
  • Benjamin
Last updated on by Warren Anderson